Make locations for detectors dir and models dir configurable. #307

ruck94301 · 2025-09-18T18:21:27Z

There are 8 assignments in LabGym code based on LabGym/detectors and LabGym/models directories.
This PR to replace those 8 with assignments based on config (which is defaults to the traditional locations (LabGym/detectors and LabGym/models).

Since the defaults are the traditional locations, this change should produce no behavior change, just a new ability to configure the locations when desired.

The changes are to 4 files:

Update config.py defaults with new detectors and models values, reflecting current usage, namely LabGym/detectors and LabGym/models. Note these are intentionally strings, not pathlib.Path objects, to match existing usage.
Enhance config.get_config to take *args.
Import config, in 3 files.
Get config values of detectors/models in the init of 6 classes:
gui_analyzer.PanelLv2_AnalyzeBehaviors needs models and detectors
gui_categorizer.PanelLv2_GenerateExamples needs detectors
gui_categorizer.PanelLv2_TrainCategorizers needs models
gui_categorizer.PanelLv2_TestCategorizers needs models
gui_detector.PanelLv2_TrainDetectors needs detectors
gui_detector.PanelLv2_TestDetectors needs detectors
Replace 8 assignment statement to use config value.
Note the existing use expects strings, not pathlib.Path objects.
So either
(a) config.get_config() should return string, not pathlib.Path
objects, or
(b) the 8 assignment statements should convert with str(), or
(c) the assignments and all other downstream manipulations should be
rewritten for pathlib.Path objects.
The least invasive path is (a).
Eliminate the_absolute_current_path module variable (3 files)

Also,
7. wrote logger.debug statements to shadow the 8 assignments. Since relevant unit tests don't exist, this provides a way to demonstrate that the assignments are executed.

Test results which show that all 8 assignments were executed:

Assignment 1 of 8 is in gui_analyzer.py
Analysis Module --> Analyze Behaviors --> Select a Categorizer ...
2025-09-18 09:34:05 DEBUG [4320867840:LabGym.gui_analyzer:gui_analyzer:253] self.model_path: '/Users/john/Public/LabGym/LabGym/models'

Assignment 2 of 8 is in gui_analyzer.py
Analysis Module --> Analyze Behaviors --> Specify the method to detect animals or objects --> Use trained Detectors (versatile but slow)
2025-09-18 10:20:03 DEBUG [4437001728:LabGym.gui_analyzer:gui_analyzer:564] self.detector_path: '/Users/john/Public/LabGym/LabGym/detectors'

Assignment 3 of 8 is in gui_categorizer.py
Training Module --> Generate Behavior Examples --> Specify the method to detect animals or objects --> Use trained Detectors (versatile but slow)
2025-09-18 10:29:57 DEBUG [4437001728:LabGym.gui_categorizer:gui_categorizer:382] self.detector_path: '/Users/john/Public/LabGym/LabGym/detectors'

Assignment 4 & 5 of 8 are in gui_categorizer.py
Training Module --> Train Categorizers
2025-09-18 09:32:32 DEBUG [4320867840:LabGym.gui_categorizer:gui_categorizer:1087] self.model_path: '/Users/john/Public/LabGym/LabGym/models'
2025-09-18 09:32:32 DEBUG [4320867840:LabGym.gui_categorizer:gui_categorizer:1089] self.path_to_categorizer: '/Users/john/Public/LabGym/LabGym/models/New_model'

Assignment 6 of 8 is in gui_categorizer.py
Training Module --> Test Categorizers
2025-09-18 09:32:35 DEBUG [4320867840:LabGym.gui_categorizer:gui_categorizer:1594] self.model_path: '/Users/john/Public/LabGym/LabGym/models'

Assignment 7 of 8 is in gui_detector.py
Training Module --> Train Detectors
2025-09-18 09:31:58 DEBUG [4320867840:LabGym.gui_detector:gui_detector:246] self.detector_path: '/Users/john/Public/LabGym/LabGym/detectors'

Assignment 8 of 8 is in gui_detector.py
Training Module --> Test Detectors
2025-09-18 09:32:26 DEBUG [4320867840:LabGym.gui_detector:gui_detector:403] self.detector_path: '/Users/john/Public/LabGym/LabGym/detectors'

Also, from __future__ import annotations

Implement configurable detectors dir and models dir, with default defined in config.py.

Also eliminate the_absolute_current_path module variable (no longer in use).

yujiahu415 · 2025-09-19T12:57:22Z

Hi John,
Have you tested the method to locate the LabGym path cross different platforms to make sure it returns the correct path? This change is critical since any issue may prevent users from accessing their trained models. When I used that 'absolute path' way, I actually tested different methods and that particular way gave me the most consistent correct results and it indeed never failed in real scenarios for all the users so far.

ruck94301 · 2025-09-19T14:17:50Z

Hi John, Have you tested the method to locate the LabGym path cross different platforms to make sure it returns the correct path? This change is critical since any issue may prevent users from accessing their trained models. When I used that 'absolute path' way, I actually tested different methods and that particular way gave me the most consistent correct results and it indeed never failed in real scenarios for all the users so far.

My testing has been only on Mac.
The functionality of the absolutepath variable is now in the two lines in config.py

 71     'detectors': str(Path(__file__).parent.joinpath('detectors')),
 72     'models': str(Path(__file__).parent.joinpath('models')),

These are actually very easy to sanity-check on any platform that we have access to. LabGym calls config.get_config() very early, and if debug logging is turned on, there is a statement that outputs the config dictionary:

% LabGym --debug
2025-09-19 08:06:00	DEBUG	[4355081728:LabGym.mylogging:mylogging:361]	Added a QueueHandler to root logger.
2025-09-19 08:06:00	DEBUG	[4355081728:LabGym.mylogging:mylogging:363]	rootlogger.handlers: [<QueueHandler (NOTSET)>]
2025-09-19 08:06:00	DEBUG	[4355081728:LabGym.mylogging:mylogging:364]	rootlogger.level: 0
2025-09-19 08:06:00	DEBUG	[4355081728:LabGym.__main__:__main__:40]	loading /Users/john/Public/LabGym/LabGym/__main__.py
2025-09-19 08:06:00	DEBUG	[4355081728:LabGym.mylogging:mylogging:363]	rootlogger.handlers: [<QueueHandler (NOTSET)>]
2025-09-19 08:06:00	DEBUG	[4355081728:LabGym.mylogging:mylogging:364]	rootlogger.level: 0
2025-09-19 08:06:00	DEBUG	[4355081728:LabGym.config:config:111]	/Users/john/.labgym/config.toml: file not found
2025-09-19 08:06:00	DEBUG	[4355081728:LabGym.config:config:230]	result:
{'anonymous': False,
 'configdir': PosixPath('/Users/john/.labgym'),
 'configfile': PosixPath('/Users/john/.labgym/config.toml'),
 'detectors': '/Users/john/Public/LabGym/LabGym/detectors',
 'enable': {'central_logger': True, 'registration': True},
 'logging_configfiles': [PosixPath('/Users/john/.labgym/logging.toml'),
                         PosixPath('/Users/john/.labgym/logging.yaml'),
                         PosixPath('/Users/john/Public/LabGym/LabGym/logging.yaml')],
 'logging_level': 'DEBUG',
 'models': '/Users/john/Public/LabGym/LabGym/models'}

Henry/Bobby, how can we do a sanity check on other platforms, for the dict values for key 'detectors' and key 'models'?

yujiahu415 · 2025-09-19T14:41:43Z

I see. Thanks for the clarification.

ruck94301 · 2025-09-19T15:03:44Z

BTW, when I was exploring PyInstaller, I was wondering if references to file would be evaluated at PyInstaller "compile" time or at LabGym execution time. Fortunately, I was able to confirm that the file references are evaluated at LabGym execution time.

ruck94301 added 22 commits July 27, 2025 16:11

PEP-8 the imports of 3 gui_* modules

9ac6fd1

Also, from __future__ import annotations

Get detectors dir and models dir from config.

a3bd0d1

Implement configurable detectors dir and models dir, with default defined in config.py.

Fix a syntax typo in config.py

39a33e7

Merge branch 'master' into relocate-data-dirs

e684749

Merge branch 'master' into relocate-data-dirs

72b46ee

Merge branch 'master' into relocate-data-dirs

ddd6185

Merge branch 'master' into relocate-data-dirs

5f11c67

Merge branch 'master' into relocate-data-dirs

101f1e4

Merge branch 'master' into relocate-data-dirs

778dd9b

Merge branch 'master' into relocate-data-dirs

8a3b76e

Merge branch 'master' into relocate-data-dirs

922e9c5

Merge branch 'master' into relocate-data-dirs

1d6bc68

Merge branch 'master' into relocate-data-dirs

3d9ee33

Merge branch 'master' into relocate-data-dirs

3eb1949

Merge branch 'master' into relocate-data-dirs

e40f158

change default detectors/models to str objects.

53755ba

Added shadow logger.debug statement to detectors & models statements.

476a2e1

Also eliminate the_absolute_current_path module variable (no longer in use).

Add args to config.get_config().

0faf8e4

Use config.get_config() with args

f30c02c

Add shadow logging statements to detectors and models assignments.

0aec085

Fix joinpath needs to be os.path.join

ddc5d9c

Merge branch 'master' into relocate-data-dirs

4fc291c

yujiahu415 merged commit 77efe6c into umyelab:master Sep 19, 2025
4 checks passed

ruck94301 deleted the relocate-data-dirs branch September 21, 2025 17:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make locations for detectors dir and models dir configurable. #307

Make locations for detectors dir and models dir configurable. #307

Uh oh!

ruck94301 commented Sep 18, 2025

Uh oh!

yujiahu415 commented Sep 19, 2025

Uh oh!

ruck94301 commented Sep 19, 2025

Uh oh!

yujiahu415 commented Sep 19, 2025

Uh oh!

Uh oh!

ruck94301 commented Sep 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Make locations for detectors dir and models dir configurable. #307

Make locations for detectors dir and models dir configurable. #307

Uh oh!

Conversation

ruck94301 commented Sep 18, 2025

Uh oh!

yujiahu415 commented Sep 19, 2025

Uh oh!

ruck94301 commented Sep 19, 2025

Uh oh!

yujiahu415 commented Sep 19, 2025

Uh oh!

Uh oh!

ruck94301 commented Sep 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants